The performance of conventional CMOS Image Sensor (CIS) is usually subject to a tradeoff between the signal-to-noise ratio (SNR) and the frame rate, both of which are directly linked to the exposure time as used in image taking. On the one hand, short exposure time allows a high frame rate and can be employed to capture fast motions, but this also leads to a low pixel SNR at low light intensity. On the other hand, long exposure time can improve pixel SNR but usually induce motion blurring in the resulting images and photodiode well saturation.
A number of computational imaging methods have been developed to address these tradeoffs. One example is the pixel-wise exposure using a digital micro-mirror device (DMD). With controlled exposure, the resulting system can operate at a slow frame rate with better SNR and dynamic range for high-speed imaging tasks. Another example is the flutter shutter technique, which can reduce motion blurring by exposing the pixels using a temporally coded shutter, instead of a continuous exposure. The added pattern can improve invertibility of the blur matrix, thereby increasing the ability to de-wrap a blurred image.
Inspired by the theory of Compressed Sensing (CS), a number of CS-based imaging techniques also emerged to improve spatial and temporal resolution of image sensors. Existing CS-based sensors use optical frontend to apply a random pixel wise exposure pattern to the focal plane of the sensor. The image sensor then samples the modulated video. These methods compress a spatiotemporal video into a single image. Using inherent sparsity in natural scenes, the video can be then recovered from the compressed image using optimization algorithms.
Previous temporal CS imaging systems have demonstrated high image quality at high reconstruction frame rate. But all the previous implementations (both CS based and non-CS based) use optical apparatus to pre-modulate the video scene before the image sensor. For example, optical exposure control can use off-chip spatial light modulators (SLM), such as digital micro-mirror devices (DMD) or liquid-crystal-on-silicon (LCOS) devices, to modulate pixel exposure prior to the sensor focal plane. Using different spatiotemporal optical masks, exposure-coded imaging can capture blur-free motion at a slow frame rate. However, the opto-mechanical apparatus for optical exposure control increases the overall system size and power consumption, thereby limiting the potential of the resulting system to be further miniaturized.